Technical Field
The present disclosure relates generally to interactive three-dimensional (“3D”) displays. More particularly, the present disclosure addresses apparatus, systems, and methods making up a display system with 3D in situ visualization that can maintain the eye's natural accommodation-vergence relationship.
Description of Related Art
Interactive 3D display systems have been the subject of a number of developmental efforts over the past 30 years. The prospect of reaching out and directly interacting with virtual content is universally intriguing and may allow for step changes in creativity and efficiency in developing models, creating art, and understanding and manipulating complex data. Several groups have pursued merging 3D interactive displays in-situ with real environments. One aim of these groups has been to enable real-time guidance for critical tasks where there is limited visibility. Another aim has been to allow for accurate and intuitive in-field visualization of complex data.
Medicine is one of the fields that stand to benefit the most from directly interactive 3D display systems with in situ visualization. Surgeons are required to carry out operations in the least amount of time and with minimal invasiveness. Understanding the layout of the patient's internal anatomy may allow surgeons to plan the shortest and most direct path for completing operations. While CT, MRI, and ultrasound scans accurately lay out a patient's anatomical information, during surgery these modalities are usually displayed on monitors out of the field of view of the patient's body. The result is that surgeons have to mentally store scanned patient data from one view and transform and apply it to the view with the patient. A few methods have been developed to provide co-location of scanned data with the patient.
Head-mounted stereoscopic displays (“HMD”) were proposed in some efforts as a solution, but these are heavy and awkward to use because the cables running to the HMD can restrict the freedom of movement of the user. HMDs are limited to displaying content at a single fixed or finite set of focal lengths. The focal length for single focal length HMDs is usually set at infinity while patient images from the display's stereo screen converge at the actual distance of the patient (usually arm's length or less). This disparity may result in accommodation-vergence conflict where the eyes converge on a plane at a certain distance but are accommodated at a plane at another distance. Breaking of the natural accommodation-vergence relationship can lead to eye fatigue and result in difficulty achieving optical fusion where left and right images no longer appear fused. One HMD has been designed with three focal lengths. In this system, software toggles between the three fixed focal lengths and infers the closest appropriate length based on the position of the user in relation to the virtual content. This solution could bring the disparity in accommodation-vergence closer in line. However, as the nature of surgery requires surgeons to arbitrarily move closer to patients for more detail and further away to establish the overall layout, there would be frequent significant disparities in the regions between the focal lengths.
Head-mounted displays are also especially prone to temporal misalignment in the imagery as a result of latency. This latency is significant during fast head movements, and in augmented reality applications the magnitude of the latency is intensified in proportion to the distance between the viewer and the subject. In medical settings, the distance between the surgeon and patient can be enough to introduce significant latency issues. Another issue with using head-mounted displays in surgical settings is that assistants are not able to observe with the surgeon the augmented graphics presented in context with the patient unless they themselves are wearing head-mounted displays, which adds additional cost and complexity to the system. Assistants are usually left to follow along on standard overhead displays, with the original disadvantage of not being able to fuse the patient data with actual anatomy.
Various additional display systems implemented to provide interactive 3D display systems with in situ visualization may use combinations of technique and/or equipment such as projection of images using semi-transparent mirrors, sensors to track the viewer's head to overlay a virtual view co-located with a subject, and stereoscopic viewing devices. Such display systems exhibit several shortcomings. For example, such display systems may result in the viewer repeatedly shifting focus between the projected image plane and the subject, which is unintuitive and could lead to, for example, blurred vision and/or inaccurate movements during surgery. Other shortcomings of such display systems may include large system footprints, reduced access to patients, weak and expensive equipment, latency in viewer movement tracking, and inducement of eye strain, fatigue, and dizziness in the viewer.